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Introduction 

Multimedia consists of: 

* multimedia data 

* interaction set 


Multimedia data: 

* multi-source, multi-type, multi-format 


Interaction set: 

* without interactions between multimedia components, 
multimedia is merely a collection of data 


Introduction 


Example: Augmented reality conference 



real objects 

virtual objects 

real speech 
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What Is Multimedia Processing? 


• Multimedia processing 

• apply signal processing tools to multimedia 
data to enable: 

» representation 
» interpretation 
» encoding 
» decoding 


Introduction 


Goals of multimedia processing: 


• Effective & efficient 

• access 

• manipulation 

• exchange 

• storage 

of multimedia content 


Multimedia Data: Input and Format 


Text and Static Data 

• Source: keyboard, speech input, optical character 
recognition, data stored on disk. 

• Stored and input character 

• Storage of text is 1 byte per char / more bytes for Unicode. 

o For other forms of data (e.g. Spreadsheet files). 

o May store format as text (with formatting) others may 
use binary encoding. 

• Format: Raw text or formatted text e.g HTML, Rich Text 
Format (RTF), Word or a program language source (C, 
Pascal, etc. 



Multimedia Data: Input and Format 

Graphics 

• Format: constructed by the composition of primitive objects such 
as lines, polygons, circles, curves and arcs. 


• Input: Graphics are usually generated by a graphics editor 
program (e.g. Illustrator). Graphics are usually editable. 


• Graphics input devices: keyboard (for text and cursor control), 
mouse, trackball or graphics tablet. 

• Graphics standards : OpenGL, PHIGS, GKS 


• Graphics files usually store the primitive assembly 


Do not take up a very high storage overhead 



Multimedia Data: Input and Format 

Images 

• Still pictures which (uncompressed) are represented as a bitmap (a grid of 
pixels). 


• Input: digitally scanned photographs/pictures or direct from a digital 
camera. 

• Input: May also be generated by programs "similar" to graphics, or 
animation programs. 


• Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale, 
Colour Map) or 24 Bits per pixel (True Colour) 


• Size: a 512x512 Grey scale image takes up 1/4 MB, a 512x512 


• 24 bit image takes 3/4 MB with no compression. 


Compression is commonly applied with the increased size of the images. 



Multimedia Data: Input and Format 

Audio 

• Audio signals are continuous analog signals 

• Input: microphones and then digitized and stored 

• CD Quality Audio requires 16-bit sampling at 44.1 KHz 

• Even higher audiophile rates (e.g. 24-bit, 96 KHz) 

• 1 Minute of Mono CD quality (uncompressed) audio 
requires 5 MB. 

• 1 Minute of Stereo CD quality (uncompressed) audio 
requires 10 MB. 


Usually compressed (E.g. MP3, AAC, Flac, Ogg Vorbis) 



Multimedia Data: Input and Format 

Video 

• Input: Analog Video is usually captured by a video camera and then 
digitized. 

• There are a variety of video (analog and digital) formats 

• Raw video can be regarded as being a series of single images 

• There are typically 25, 30 or 50 frames per second 

• Typical PAL digital video (720 576 pixels per colour frame) 


High Definition video on Blu-ray (up to 19201080 = 2 Megapixels per 
frame) Digital video clearly needs to be compressed for most times 



Multimedia Data Compression 


How can we compress data? 

Lossy vs Lossless : 

o Lossless : Ideal (e.g. zip, unix compress) not good enough for 
MM data! 

o Lossy:Throw away nonessential (perceptually less relevant) 
parts of the data stream FILTER the data somehow. Examples: 
MP3, JPEG, MPEG Video 



Video Compression 

Raw rate: 

- images: 24 or 30 per second. 

- Size: 1024 x 1024 = 1 M pixels 

- pixel encoding 24 bit 

- Rate: 576 or 900 Mbps !!! 

Compression schemes (many): 

- MPEG 1 (1.5 Mbps) CD quality 

- MPEG 2 (3-6 Mbps) DVD quality 

- Motion JPEG 

- H.261 (for ISDN) 


Variable rate compression 



What is multimedia system? 


A Multimedia System is a system capable of processing 
multimedia data and applications. 


It is characterized by the processing, storage, generation, 
manipulation and interpretation of the Multimedia 
information. 



Multimedia System Characteristics 


A Multimedia system has four basic characteristics: 

• Multimedia systems must be computer controlled. 

• Multimedia systems are integrated. 

• The information they handle must be represented 
digitally. 

• The interface to the final presentation of media is usually 
interactive. 



Challenges for Multimedia Systems 


* Distributed Networks 

* Temporal relationship between data 

- Render different data at same time — continuously. 

- Sequencing within the media playing frames in correct 
order/time frame in video. 


Synchronization — inter-media scheduling e.g. Video and 
Audio conversation. 



Key Issues for Multimedia Systems 


The key issues multimedia systems need to 
deal with include: 


-How to represent and store temporal 
information 


-How to strictly maintain the temporal 
relationships on play 



Key Issues for Multimedia Systems 


back/retrieval 

What process are involved 

Data has to be represented digitally 

Conversion, Sampling etc. 

Large Data Requirements — bandwidth, 
storage, Data compression is usually 
mandatory. 



Desirable Features for a Multimedia 

System 

According to the mentioned challenges, the 
following feature is required for a Multimedia 
System: 

-Very High Processing Power — needed to deal 
with large data processing and real time 
delivery of media. 



Multimedia System: Required Features 

Special Hardware/Software needed 


Data Representations — File Formats that support multimedia should be 
easy to handle yet allow for compression/decompression in real-time 


Efficient and High I/O —input and output to the file subsystem needs to 
be efficient and fast. Needs to allow for real-time recording as well as 
playback of data 

Special Operating System —to allow access to file system and process 
data efficiently and quickly. 


Storage and Memory — large storage units. 


Network Support — Client-server systems -Software Tools — user 
friendly tools needed to handle media, design and develop applications 
to deliver media. 



Multimedia System: Components 

Required Hardware and Software components for a multimedia system: 

□ Capture devices 

□ Video Camera, Video Recorder, Audio Microphone, Keyboards, mice, 
graphics tablets. 

□ Storage Devices — Hard disks, CD-ROMs, DVD-ROM, etc. 

□ Communication Networks — Local Networks, Intranets, Internet, 
Multimedia or other special high speed networks 

□ Computer Systems — Multimedia Desktop machines, Workstations. 

□ Display Devices, quality speakers, HDTV, monitors, Colour printers 
etc 



Impact of Multimedia 


Users (society) demand 

• increased mobility 

• ease-of-use 

• personal customization 

• device flexibility 

• high level of collaboration with peers 


Devices mutate and become 

• multi-functiona , not specialized 

• effortlessly portable, not stationary 

• ubiquitously networked, not isolated 


Impact of Multimedia 


Multi-functional devices must 


• browse internet 

• entertain 

• be easy-to-use 


• facilitate many 
types of workflow 

• manage user's time 


Customization 

• personalization (themes, preferences) 


Networked 

• capable of connecting to many different networks 


Impact of Multimedia 


Convergence 


Technologies which were totally unrelated 10 
years ago are now unified under the concept of 
multimedia 


Impact of Multimedia 


• Example: Cellular Phones 


Primary Consumer Use: 

• Wireless Telephony 

J 








CONVERGED USES 
f • personal organizer 

• INTERNET BROWSER/EMAIL 

• Entertainment (Mp3, Radio) 


• PAGER/MESSAGING (SMS) 

• VIDEO/STILL CAMERA 





Overall Impact of Multimedia 

• Demands 

• functionality 

• consumption of many media types 

• connectivity 

• portability, etc. 


• Result 

• highly complex devices 

• push towards dense circuitry 

• multimedia devices become ubiquitous 

• devices generate multimedia data (including 
images, video, audio) 


Multimedia Application Classes 

Typically: 

- sensitive to delay, but 

- can tolerate packet loss 

Data contains audio and video content 
("continuous media"), three classes of 
applications: 

- Streaming 

- Unidirectional Real-Time 

- Interactive Real-Time 


Application Classes 


Streaming 

— Clients request audio/video files from servers and 
pipeline reception over the network and display 

— Interactive: user can control operation (similar to 
VCR: pause, resume, fast forward, rewind, etc.) 

— Delay: from client request until display start can 
be 1 to 10 seconds 


Application Classes 


Unidirectional Real-Time: 

— similar to existing TV and radio stations, but delivery 
on the network 

— Non-interactive, just listen/view 

Interactive Real-Time : 

— Phone conversation or video conference 

— More stringent delay requirement than Streaming and 
Unidirectional because of real-time nature 

— Video: < 150 msec acceptable 

— Audio: < 150 msec good, <400 msec acceptable 


Multimedia Applications 



Scalable Video 
Streaming 



GPS Navigation 










Multimedia Applications 



E-commerce 


Cellular 







Multimedia Applications 


• More Specific Examples 


♦ MPEG-4, 7, 21 

♦ JPEG-2000 

♦ MP3 & PERCEPTUAL 

CODING 


Multimedia Storage 
❖ Video-on-demand 
•> Digital Cinema 
•> Authentication 


• Multimedia Application Goals 

• improve interpersonal communication 

• promote understanding of ideas 

• allow interactivity with media 

• increase accessibility to data 


Content Based Image Retrieval (CBIR) 


EXAMPLE: GENERAL PHOTOGRAPHY 


• Polaroid filed for bankruptcy 

• has digital killed film? if so, why? 


> SNAPSHOT 
PREVIEWS 



> EASY SHARING 
VIA INTERNET 


> 


> 


MEMORY 

REUSABLE 

printer 



tech 


> CHEAP & DENSE 
STORAGE 

> EFFECTS & 
PROCESSING 


• RESULT: DIGITAL MEDIA FLOOD 

• HOW DO WE COPE, TRACK, ORGANIZE IT ALL? 







CBIR: Motivation 


• Device Function Convergence 

• Data Rapidly Generated By Many Devices 

• Internet Acts As Global Transport 

• Data Consumed By Devices On Demand 

• Multimedia Data Needs To Be 

• Efficiently Stored 

• Indexed Accurately 

• Easily Retrieved 


CBIR: IS... 


Part of Multimedia Indexing 

• Images (2-d Space-dependent Signals) 

• Video (Time-varying Image Set) 

• Audio (1-d Time-dependent Signals) 

• TEXT (E.G. Book Index, Search Engines) 

Computer based 
Highly automated 


Difficult to do properly 


CBIR Application Areas 



• MEDICAL IMAGING 

ART/CULTURAL HERITAGE • 

• DESIGN/VISUAL ARTS 

ENTERTAINMENT (FILM, TV) • 

• INDUSTRY (LOGO MANAGEMENT) 

GOVERNMENT (E.G. MUGSHOTS) • 








MPEG-7: SCOPE 
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CBIR: SUMMARY 

• Started from multimedia flood 

• F ext too simple and laborious 

• Systems work decently in vitro 

• query by shape, color, texture, example 

• Shortcomings 

• need relevance feedback & perceptual 

• hybrid queries difficult to create 

• semantic gap needs to be bridged 

• Motion pictures expert group (MPEG-7): 
important development 


Summary 

Multimedia processing 

• results from multimedia explosion 

• users demanding more from devices 

• devices are converging 

Content based image retrieval 

• necessary to track visual sea of data 

• good capabilities, but w/ shortcomings 

• perceptual/subjective issues 

• relevance feedback 

• distributed concepts becoming critical 

MPEG-7 

• aimed at standardizing descriptions 

• radically different than previous mpegs 

• DDL is an extension of xml schema 

• applicable to all multimedia data 



Any Questions!! 


